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This  is  the  second  quarterly  status  report  on  a program  for  RECOGNITION 
TECHNOLOGY  FOR  A SMART  SENSOR,  being  conducted  by  the  Westinghouse  Systems 

✓ 

Development  Division  for  the  Computer  Science  Center,  University  of  Maryland. 
The  program  consists  of  three  phases,  as  follows: 


Phase  — Task  and  Technology  Review  (3  months)  *j 


Phase  Pi  -Algorithm  Selection  and  Test  (9  months)^* 
Phase  IT? - Hardware  Development  (9  months)* 


This  report  covers  the  first  3 months  of  the  Phase  12  effort..  The  report 
was  prepared  by  Mr.  Thomas  Willett  and  Dr.  Nathan  Bluzer  of ^Nestinghouse . 
The  Westinghouse  program  manager  is  Dr.  Glenn  E.  Tlpdele'T 


iring  the  quarter  six  meetings  were  held  between  members  of  the  Maryland 
id  Westinghouse  teams.  Mr.  John  Dehne,  NVL  program  manager,  attended  some  of 
the  meetings  as  well  as  holding  a Symposium  on  Smart  Sensor  Technology  at  NVL. 
The  Maryland-Westinghouse  team  made  a presentation  at  that  meeting  as  well  as 
at  the  semi-annual  DARPA  Symposium,  chaired  by  Maj . David  Carlstrom,  on  Image 
Understanding,  held  at  the  University  of  Maryland  on  October  13  and  14.  Westing- 
house received  further  guidance  from  Maj.  Carlstrom  and  Mr.  Dehne  in  the  per- 
S 

ftermance  of  this  contract  from  these  Symposia. 


lestinghouse  is  concentrating  on  the  hardware  implementation  of  the 
Maryland  algorithms  for  the  focal  plane  and  treating  them  as  a system. 


Clearly,  it  is  desirable  to  Incorporate  an  auto  cueing  system  into  as  small 
and  cheap  a box  as  possible.  Implementation  on  the  focal  plane  in  compatible 
structures  is  desirable  for  these  reasons  and  is  the  thrust  of  Second  Generation 
FLIR  work. 


\ 
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Westinghouse  hopes  to  determine  system  partitioning  by  first  attempting 


to  put  as  much  of  the  system  on  the  focal  plane  as  possible,  starting  with 
the  lower  level  functions  first. 


1.0  FOCAL  PLANE  PROCESSING 


The  focal  plane  of  a Smart  Sensor  contains  the  array  of  sensor  elements, 
e.g.,  infrared  detectors.  Each  element  In  the  array  Is  divided  in  two  parts. 
The  front  part,  facing  the  scene,  is  the  Infrared  detector  and  the  back  part 
is  an  analog  CCD  which  holds  a charge  proportional  to  the  detected  signal 
strength  of  the  scene  as  shown  in  Figure  1-1.  For  FLIRS,the  focal  plane  is 
located  in  a dewar  flask  which  is  kept  at  cryogenic  temperatures  of  20* -50s 
Kelvin.  The  leads  to  the  focal  plane  electronics  come  through  the  dewar  and 
therefore  act  as  a heat  source.  Focal  plane  electronics  are  an  additional 
heat  source  and  must  be  designed  for  operation  at  cryogenic  temperatures. 


Figure  1-1.  Side  View  of  an  IR  Detector  - CCD  Array 

Although  the  array  is  shown  as  one  dimensional  in  Figure  1-1,  it  is  a 
two  dimensional  array  with  rows  and  columns  of  detector-CCD  elements.  Now, 
we  can  limit  the  field  of  view  of  each  element  in  a row  so  that  each  element 
only  sees  a portion  of  the  scans.  If  we  move  the  entire  array  upward  by  an 
amount  equal  to  the  element  field  of  view  the  second  row  will  see  the  same 
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portion  of  the  scene  that  the  first  row  saw.  This  same  effect  can  be  achieved 
by  sweeping  a mirror  showing  the  scene  vertically  down  the  array.  If  the 
detected  charge  in  row  1 elements  is  shifted  down  to  row  2 at  the  same  time  that 
the  identical  portion  of  the  scene  is  swept  to  row  2 elements  (as  shown  in 
Figure  1-2),  we  can  enhance  the  signal  to  noise  ratio.  The  signal  strengths  add 
coherently  and  the  noise  adds  incoherently  in  the  same  fashion  as  repeated 
radar  returns  on  a target.  The  CCD  portion  of  the  array  element  is  the 
mechanism  for  adding  the  signals  and  noise  . This  is  called  Time  Delay 
Integration  and  has  been  referred  to  as  signal  processing  on  the  focal  plane. 

Note  in  Figure  1-2 , that  the  image  is  shifted  from  the  focal  plane  at  the  rate 
of  one  aegapixel/sec  but  the  PI/SO  (parallel  in/serles  out)  shift  register  receives 
a line  of  image  at  a time. 
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Figure  1-2  Time  Delay  Integration 
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For  the  purpose  of  this  work  we  shall  be  enlarging  the  conventional 
definition  of  signal  processing  on  the  focal  plane  to  include  cueing  functions, 
and  there  are  several  approaches  to  accommodating  cueing  functions  on  the 
focal  plane.  One  way  is  to  simply  enlarge  the  focal  plane  chip  and  place 
several  tiers  of  processing  between  the  last  row  of  TD1  processing  and  the 
parallel  in/series  out  shift  register.  Another  possible  approach  is  to  perform 
cueing  functions  as  the  charges  are  being  generated  and  passed  down  the  array; 
here  there  is  an  opportunity  to  address  the  x-y  position  of  a particular  pixel,  I 
but  the  integration  Is  interrupted.  In  this  quarter,  we  used  the  first  approach 

1 

only  because  it  allowed  us  to  attack  the  mathematical  and  logic  functions 
required  by  the  cueing  functions  directly. 

The  number  of  cueing  functions  placed  on  the  focal  plane  will  depend 
on  the  tradeoff  among  the  availability  of  spr _e  on  the  focal  plane,  yields 
fcr  chips  of  small  size  and  complexity,  chip  performance  at  cryogenic 
temperatures,  and  the  effects  of  a number  of  leads  penetrating  the  dewar. 
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2.0  SYSTEM  FLOW 


This  section  describes  a preferred  set  of  algorithms  developed  by 
Maryland  which  tentatively  comprise  the  cueing  system.  A system  flow  chart 
is  shown  in  Figure  2-1.  A description  of  data  flow  and  storage  requirements 
is  included. 
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Figure  2-1.  System  Flow  Chart 


2 . 1 Algorithms 


The  order  and  a short  description  of  each  algorithm  follows. 

2.1.1  Median  Filter 

The  Median  Filter  is  the  first  algorithm  performed  and  acts  to  extract 
the  median  gray  level  from  a 5 x 5 array  of  pixels,  as  shown  in  Figure  2-2, 
and  to  place  that  median  value  in  the  pixel  location  a33-  The  median  value  is 
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Figure  2-2.  Median  Filter 
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defined  as  the  13th  gray  level  value  in  an  ordering  of  the  25  gray  levels  by 
magnitude,  counting  from  either  end.  The  Median  Filter  acts  as  a moving  5x5 
window  in  that  having  obtained  m^,  column  1 (a^,  a21’  a51^  remove<*  ^rom 

the  ordering  and  column  6 (a^g,  a2g»  ***  a56^  ac^e<*  with  the  accompanying 
reordering  and  the  13th  gray  level  value,  m12,  is  obtained,  and  placed  in  the 
a14  Pos^bion.  Having  completed  an  entire  line  of  image,  row  1 is  dropped,  row 
6 is  added  and  the  process  is  repeated.  Maryland  has  chosen  to  ignore  algorithm 
computations  along  the  image  edges  for  now. 

2.1.2  Threshold  Algorithm 

The  Threshold  Algorithm  first  computes  an  operator,  OP  = max  ^ | A-B  | , 

| C-D | ^ based  on  four  regions  A,  B,  C,  and  D,  each  of  which  consists  of 
4x4  pixels  as  shown  in  Figure  2-3  and  the  composite  as  seen  in  Figure  2-4. 
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Figure  2-4.  The  Composite  of  A,  B,  C,  and  D 


where : 

llo 

A = £ ai  , a1  is  a pixel  in  A 

i-  i 

1 4 

B ■ b , b.  is  a pixel  in  B 

i*i  3 3 

lb 

C = ^ is  a pixel  in  C 

•in 

& 

D = } d , d is  a pixel  in  D 

m m 

**S| 


The  operator  OP  works  as  a moving  window  in  that  the  leftmost  column 


is  removed  from  each  region  and  the  next  right  column  is  added  to  each.  The 
operator  result  is  placed  in  the  pixel  location  marked  by  an  X. 


2. 1.2.1  Two  Dimensional  Histogram 

At  this  time,  a median  value  and  a gradient  value  have  been  computed  for 
each  pixel  location  in  a 64  x 64  pixel  window.  This  is  the  size  of  the  moving 
window  which  Maryland  will  use  to  look  for  targets.  A two  dimensional  histogram 
is  formed  with  median  gray  level  on  one  axis  and  gradient  value  on  the  other  as 
shown  in  Figure  2-5. 
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Figure  2-5 . Histogram 


Then  a gradient  value  corresponding  to  (e.g.)  802  of  the  gradient  population,  grad.., 

80 


is  found  and  the  median  values  whose  corresponding  gradient  value  are  greater 
than  gradgQ  are  averaged.  This  subset  of  the  histogram  is  shown  as  the  cross 
hatched  section  in  Figure  2-5.  The  average  of  this  subset  of  median  values, 
med,  is  then  used  to  threshold  the  original  64  x 64  image  of  gray  level  values. 


2.1.2  Binary  Operator 


As  stated,  the  average  of  the  subset  of  median  values  is  used  to  threshold 
the  64  x 64  image  of  original  gray  level  values.  Those  gray  level  values  which 
are  equal  to  or  exceed  med  are  set  equal  to  1,  and  those  values  which  are  less 
than  med  are  set  equal  to  zero.  The  result  is  a binary  image. 


2.1.3  Noise  Region  Filtering 


The  Noise  Region  Filtering  Algorithm  acts  to  remove  noise  bursts  which 
have  survived  the  previous  thresholding  but  are  small  in  area  and  isolated. 
This  algorithm  is  a shrink  and  expand  process  with  the  following  logic.  The 
shrink  test  may  be  stated  as 
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q'(c)  ■ ( 1 i£  c ■ 1 and  a+b+d+e*4 
I 0 otherwise 

where  the  arrangement  of  pixels  e through  e Is 

s 

bed 
e \ 

The  expansion  test  may  be  stated  as 

<| ; (c)  ” llifc«lora  + b + d + e^O 
( 0 otherwise 

and  the  arrangement  of  the  pixels  Is  the  sane  as  above.  Each  test  is  repeated 
three  tines. 

Having  described  each  algorithm  to  a point  necessary  for  hardware 
implementation , the  next  step  Is  to  see  how  the  algorithms  Interface  with 
each  other  and  the  focal  plane  structure. 


2 . 2 Data  Flow 


2.2.1  Obtain  Image  Data  for  Median  Filter  and  Gradient  Operator 


From  preceding  discussions  of  the  Median  Filter  and  Gradient  Operator, 
it  was  seen  that  the  Median  Filter  required  5 lines  for  processing.  Similarly, 
the  Gradient  Operator  required  8 lines.  If  we  superimpose  the  Median  Filter 
and  the  Gradient  Operator  so  that  their  respective  results  would  represent 
the  same  pixel  location,  the  combination  is  seen  in  Figure  2-6. 
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Figure  2-6.  Super -Posit ion  of  Median  and  Gradient  Operators 


It  is  noted  from  Figure  2-1  that  the  original  image  will  be  used  later 
in  a thresholding  operation  to  produce  a binary  image;  this  means  the  64  x 64 
image  must  be  stored  somewhere  while  the  Median  Filter  and  Gradient  Operator 
are  working  through  the  image.  Coupled  with  the  8 line  requirement  for  the 
Gradient  Operator,  a rough  cut  can  be  attempted  at  a focal  plane  configuration 
as  seen  in  Figure  2-7. 
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Figure  2-7.  Focal  Plane  Configuration 

The  uppermost  CCD  register  is  a parallel  in,  series  out  (PI/SO)  register,  and 
the  other  CCD  registers  are  all  series  in,  series  out  (SI/SO)  so  that  the  8 
lines  are  delayed  in  a serpentine  delay  line.  The' output  of  the  serpentine 
delay  line  would  go  to  a storage  memory  where  the  original  image  is  delayed. 

The  next  step  is  to  examine  how  the  Individual  elements  will  be  nondestructively 
read  from  the  circulating  memory  in  order  to  form  the  Median  Filter  and  the 
Gradient  operator.  Consider  Figure  2-8,  where  the  individual  pixels  are  shown 
as  located  in  the  image  (2-8a)  and  as  found  in  the  serpentine  memory  (2-8b) . 
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The  region  A (4  x 4)  Is  composed  of 
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and  the  sum.  A,  Is  composed  of  all  16  terms.  These  16  terms  can  be  obtained 
from  the  serpentine  memory  by  nondestructive  readout  from  these  positions  as 
shown  in  Figure  2-6  . Similarly,  the  region  C (4  x 4)  is  composed  of 
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and  the  sum,  C,  is  composed  of  all  16  terms.  These  terms  are  also  obtained 
from  the  serpentine  memory  by  nondestructive  readout  from  the  positions  shown 
in  Figure  2-8.  The  nondestructive  readout  positions  are  also  shown  for  the 
sums  B and  D. 

The  Median  Filter  calculation  requires  the  block  of  numbers 
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2-8b.  Pixel  Locations  in  Serpentine  Memory  1030  v ,0 


Figure  2-8 . Focal  Plane  Registers 
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and  their  nondestructive  readout  positions  are  shown  In  Figure  2-8  and  designated 
by  M. 
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Suppose  all  the  above  readouts  have  been  achieved  and  the  Median  Filter 
and  Gradient  Operator  have  been  computed  for  pixel  location  a^.  It  is  now 
desired  to  compute  these  quantities  for  pixel  location  a^  by  using  the  same 
readout  positions  within  the  serpentine  memory.  This  is  equivalent  to  shifting 
A,  B,  C,  D,  and  M by  one  column  and  actually  shifting  the  entire  serpentine 
memory  one  pixel  location.  The  resulting  readout  positions  for  pixel  location 
a^  are  shown  in  Figure  2-9.  In  a similar  fashion,  the  Median  Filter  and 
Gradient  Operator  can  be  computed  for  the  pixel  locations  in  the  4th  row,  i.e., 

a44’  a45*  a46  a636  w^en  t*ie  r*8ht  edge  of  D coincides  with  the  edge  of 

the  Image.  Consider  now  the  handling  of  the  next  line  as  shown  in  Figure  2-10. 

Assume  that  the  first  line  of  the  image  has  been  shifted  in  parallel 
into  the  uppermost  register  of  the  serpentine  memory.  At  the  rate  of  1 
megapixel /second,  this  line  is  shifted  out,  in  series,  into  the  second  register 
of  the  serpentine  memory.  At  the  point  where  the  uppermost  register  is  empty, 
the  next  line  of  image  has  been  processed  by  the  TDI  array  and  is  shifted,  in 
parallel,  into  the  emptied  register  and  so  on  until  the  8 registers  are  filled. 

Now  the  8th  image  line  has  just  been  received  and  the  serpentine  will  shift 
640  positions  (the  examples  showed  only  9 pixels  per  line)  before  the  upper 

■ 

register  is  emptied  and  the  next  image  line  is  received.  When  this  occurs, 
the  uppermost  register  will  contain  the  9th  row  of  the  image,  the  next  register 
will  hold  the  8th  image  row,  and  so  on  with  the  last  register  holding  the  2nd 
row  of  the  image.  This  is  equivalent  to  dropping  regions  A,  B,  C,  D,  and  M 
down  the  image  a row  at  a time. 
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Figure  2-ii.  Details  of  Serpentine  Delay  Line 
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2.2.2  Parallel  Extraction  of  Median  Filter  and  Gradient  Operator 


In  the  previous  discussions,  we  have  implicitly  assumed  that  a single 
Median  Filter  and  Gradient  Operator  device  would  process  the  data.  This  may 
not  be  the  case  because  either  the  CCD  Implementation  is  not  capable  of 
1 megapixel/sec.  or  a higher  data  rate,  e.g.,  10  megapixels /sec.  is  desired. 
It  Is  appropriate  to  consider  a simple  example  of  a parallel  configuration. 


The  Image  comes  Into  the  serpentine  memory  a line  at  a time.  Assuming 
there  are  640  pixels  per  line,  ten  windows  can  be  formed  simultaneously 
(Maryland  is  working  with  a 64  x 64  window  at  present).  The  sensor  array 
need  not  be  divided  into  ten  sections,  but  the  PI/SO  register  is  divided. 
Each  section  may  have  its  own  serpentine  memory  and  certainly  it  will  have 
its  own  Median  Filter  and  Gradient  Operator.  Figure  2-12  shows  such  an 
arrangement.  The  memory  used  to  hold  the  original  image  would  be  divided  in 
a similar  fashion  as  shown  in  Figure  2-13  to  maintain  the  data  rates. 
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Figure  2-12.  Parallel  Organization  of  Focal  Plane 
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Figure  2-13.  Memory  Organization  for  Original  Image 


3.0  Hardware  Implementation 


In  the  prior  sections,  we  have  defined  focal  plane  processing  and  discussed 
the  data  flow  through  the  system , thereby  gaining  an  idea  of  the  storage  data 
handling  and  speed  requirements  of  the  system.  In  this  section,  we  shall 
discuss  specific  hardware  techniques  to  obtain  the  data  handling  required  and  P^rfJ 
the  algorithms. 


3.1  Serpentine  Delay  Line 
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obtained  from  the  serpentine  memory.  Further,  the  CCD  technology  to  non- 
destructlvely  readout  and  form  sums  was  discussed  in  Section  4.5  of  the  First 
Quarterly  Report.  Then,  the  focal  plane  is  as  shown  in  Figure  3-1, 


The  formation  of  the  serpentine  delay  line  and  the  nondestructive  readout 
features  as  implemented  in  CCD  technology  were  discussed  in  Section  4.5  of  the 
First  Quarterly  Report. 

3.2  Threshold  Algorithm 
3.2.1  Gradient  Operator 

The  Gradient  Operator  as  defined  by  Maryland  is  G * max  I A-B  | , | C-D  |*^ 
where  A,  B,  C,  D are  each  the  sum  of  16  pixel  gray  levels  whose  geometrical 
arrangement  was  described  in  Sections  2.1.2  and  2.2  of  this  report. 

A,  B,  C,  and  D can  be  redefined  as 
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Figure  3-1.  Focal  Plane  and  Gradient  Operator 


and  we  shall  be  concentrating  on  the  contents  of  the  box  labelled  Gradient 
Operator.  Utilizing  our  redefinition  of  A,  B,  C,  and  D as  , B^,  C and 
, the  Gradient  Operator  may  be  redefined  as 


lAij  ‘ ®ij I lf  I Aij  " Bij  I M Cij  ' Dij  | 

I Cij  " °ij  | lf  I Cij  " °ij  I - I A1J  ‘ Bij I . 


The  key  CCD  technique  to  achieve  G^  is  subtraction  using  the  potential 
Equilibration  Method  which  does  not  correspond  to  conventional  subtraction. 


M[ 


Consider  the  structure  in  Figure  3-2  with  the  two  signal  inputs  A^,  and 
the  diffusion  diode  input  D^. 


A;;  B. 

')  IJ 


Dj 


ill: 


Transfer 

Gate 
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Figure  3-2.  CCD  Subtraction  Block 


Using  potential  equilibration , the  charge  injected  via  Din  and  retained  in  the 
CCD  well  under  the  gate  will  not  equal  zero  only  if  | ( is  greater  than 

| | . Mathematically,  the  amount  of  charge  produced  in  the  CCD  by 

subtracting  from  B^  can  be  expressed  as  follows: 


K<BiJ-Vfor  Kjl<  I Enl 


<ij 


£°r  IbdI  < lAul 


where  K is  a constant.  The  aforementioned  subtraction  operation  can  be 
represented  by  a module  shown  in  Figure  3-3. 


'ij 


> Qij 
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Figure  3-3.  CCD  Subtraction  Module 
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If  two  of  the  nodules  In  Figure  3-3  are  placed  In  parallel  and  the  Inputs 
(Alj.  By)  to  each  are  commuted,  one  of  the  outputs  will  not  be  equal  to  aero, 
as  In  Figure  3-4. 


I A B I 
'I  'I 
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Figure  3-4.  CCD  Absolute  Value  Operator 


A sum  of  the  two  outputs  and  will  yield  the  absolute  value  of  A^  - . 

A similar  operation  performed  on  C^  and  will  yield  the  absolute  value  of 
the  difference  between  C^  and  . 

The  largest  absolute  difference,  i.e.,  fA^  - ( or  j C^  - | t 

can  be  obtained  if  we  make  use  of  a CCD  non-destructive  readout  module  (CCD 
NDRO)  and  form  an  assembly  shown  in  Figure  3-5.  The  modules  labelled  by  ABS 
represent  the  diagram  shown  In  Figure  3-4.  The  output  from  ABS  module  3 will 
be  equal  to  (lA±j  - B^l  - I C^  - D±J  | ) or  (ic^  - D±j|  -lA±j  - B±J  |) 

depending  on  which  quantity  is  positive.  If  we  add  to  the  output  of  ABS 
module  3 the  values  of  ] A^  - B^  | and  | C^  - D^  | , it  can  be  shown  that 
the  output  will  equal  twice  the  larger  of  the  two  inputs  | A^  - B^  j and 
l C^j  - D^  | . Mathematically  this  can  be  shown  if  we  examine  all  the 
possible  outputs  from  ABS  module  3 summed  with  inputs  j A^  - B^\  and  Jc^  - 
D^  | . The  results  are  shown  in  Figure  3-6,  and  illustrate  that  the  output 
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lAij*B jj | < JCjj  - Djj  J Cjj  - Djj  J < Ajj  - Bj 


tarDij|- 


|CjjDjj|  +jAjj-Bjj 


2 I Cjj  - Djj 


Ai| " Bij'l  |Cij  'Dii 


Cij  - Djj 


Aij  ‘ Bij 


2 Ajj  - Bjj 
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Figure  3-6.  Outputs 
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3.2.2  Two  Dimensional  Histogr 


A histogram  of  gradient  values  can  be  formed  in  the  same  way  as  that  for  the 
median  value  histogram  and  the  GRAD^q  value  found.  However,  In  shuffling  the 
gradient  values  through  the  histogram,  It  Is  difficult  to  keep  track  of  the 
corresponding  median  values.  One  approach  is  to  form  the  histogram  of  gradient 
values,  and,  at  the  same  time,  store  the  gradient  values  and  corresponding 
median  values  In  two  separate  memories.  Then,  find  GRADg^  from  the  histogram, 
and  use  It  to  threshold  a pixel  by  pixel  read  out  of  the  gradient  memory.  At 
the  same  time,  the  median  value  memory  Is  read  and  those  median  values  whose 
corresponding  gradient  exceed  GRADg^  are  kept  and  averaged  as  in  Figure  3-7. 

This  approach  Implies  a storage  capacity  for  a histogram  and  two  Images  which 
are  each  64  x 64. 


Figure  3-7.  Flow  of  Two  Dimensional  Histogram 
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Another  approach  involve*  a RAM  structure  to  fora  the  two  dimensional 
histogram.  Recall  that  the  median  values  have  been  converted  to  binary  and 
when  the  gradient  histogram  Is  formed,  the  gradient  values  will  also  be  binary 
numbers.  Suppose  that  they  will  each  be  represented  by  5 bits  so  that  each 
varies  from  0 to  31.  If  the  gradient  and  median  values  are  catenated  to  form 
a single  10  bit  address,  the  range  of  possible  addresses  is  1024,  and  the  RAM 
is  1024  addresses  long.  Suppose  further  that  the  maximum  number  of  occupants 
of  each  gradient  median  value  location  is  64,  then  the  RAM  Is  1024  x 6 bits 
wide.  In  constructing  a two  dimensional  histogram,  the  RAM  must  be  read.  Its 
contents  incremented  by  1,  and  then  written  back  Into  the  same  location. 

Here  we  are  talking  about  more  than  one  RAM  and  possibly  a cluster  of  4-6  to 
obtain  length  and  width,  each  RAM  at  least  a 22  pin  pack  which  Is  incompatible 
with  focal  plane  processing. 


3.3  Median  Filter 


In  order  to  obtain  the  median  value  from  a set  of  numbers.  It  Is  necessary 
to  obtain  an  ordering  of  their  numerical  magnitudes.  This,  of  course.  Is  equiva- 
lent to  a histogram.  One  approach  to  this  problem  of  forming  a histogram  Is  to 
convert  each  analog  number  to  a binary  one  but  represent  it  by  a "thermometer 
code",  i.e.,  the  charge  level  corresponding  to  a gray  scale  pixel  magnitude 
Is  represented  by  a vertical  column  of  l's  as  shown  in  Figure  3-8. 


Charge 

Level 


Pixel  Number 
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Figure  3-8  . Thermometer  Scale 


Before  discussing  the  mechanics  of  forming  the  histogram,  the  first  order  of 
business  Is  the  CCD  A/D  converter. 

3.3.1  CCD  A/D  Converter 

This  CCD  structure  will  show  how  an  A/D  operation  can  be  achieved  on 
the  IR  focal  plane  at  cryogenic  temperatures.  The  key  element  in  the  A/D 
converter  Is  a CCD  which  exhibits  very  low  voltage  offsets.  Such  a device 
is  needed  to  generate  the  different  voltage  levels  required  to  form  parallel 
mode  A/D  converters.  These  CCD  input  voltage  levels  cannot  exhibit  DC  offset 
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nonuniformities.  These  ere  found  in  conventional  CCD's  and  are  attributed  to 
threshold  nonuniformities.  Previous  work  in  this  area  Involved  a parallel  In/ 
series  out  CCD  with  parallel  threshold  variation.  This  CCD  employing  compensated 
direct  Injection  (CDI)  can  be  employed  for  A/D  operations  according  to  the 
following  procedure.  Each  1th  parallel  input  of  the  GDI -CCD  can  be  thought  of 
as  a common  gate  MOS  FET  with  a virtual  drain  as  seen  In  Figure  3-9.  The  net 
current  entering  the  CCD  ith  Input  node  ICCD(i)  is  equal  to  the  difference 
between  IR(i)  - Ig  ■ I^CD(i)  only  if  the  ith  reference  current  IR(i)  is 
larger  in  magnitude  than  the  signal  current  Ig. 


-vref  (i> 


CCD  Input 
Node 


'CCD*  ' * 


CDI  Gate 


Virtual 

Drain 


Transfui 

Gate 
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Figure  3-9.  CCD  Input  Structure 

Whenever  IR(i)  is  less  than  Ig  the  current  ICCD(i)  entering  the  CCD  is  zero. 
Mathematically  the  amount  of  current  entering  the  CCD  virtual  drain's  well  as 


a function  of  1R(1)  and  Ig  can  be  expressed  as 


ZCCD(i) 


-XR(i)  * Is  for  eiR(i)  ♦ Is)  > 0 


for  (-IR(i)  lg)  < 0. 


The  amount  of  charge  in  any  given  CCD  virtual  drain  will  equal  the  product  of  the 


integration  time  T and  the  current  value  defined  by  the  equation  for  Igd/i)* 
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Given  sufficient  Integration  time  (T),  the  virtual  drain's  well  will  be 

! 

completely  full  of  charge  or  empty  depending  on  the  external  conditions  operated 
on  by  the  above  equations.  Overflow  of  the  accumulating  charge  in  any  one  of 
the  i_  virtual  drains  of  the  A/D  converter  can  be  prevented  if  the  draining  port 
adjacent  to  each  virtual  drain  is  Incorporated  into  the  A/D  structure.  The 
draining  port  can  be  made  from  a single  gate  and  a drain  diffusion  bus.  The 
potential  on  the  overflow  gate  can  be  adjusted  to  a level  equal  to  the  charge 
potential  in  a filled  virtual  drain.  Any  additional  charge  entering  the  virtual 
drain  will  spill  into  the  drain  bus  via  the  channel  created  by  the  gate  of  the 
overflow  structure.  The  presence  or  absence  of  charge  in  the  virtual  drain's 
well  represents  a binary  signal  which  signifies  whether  the  signal  voltage 
| Vg  | is  respectively  less  or  greater  than  the  reference  voltage  | VRej(i)| 
Summarizing,  the  structure  shown  in  Figure  3-8  has  a transfer  function  binary 


in  nature  which  gives  a one  (zero)  output  Q(i)  when  the  JvRe^(i)j  voltage  is 
less  (greater)  than  the  | Vg  | signal  voltage. 

Consider  an  assembly  of  the  structures  shown  in  Figure  3-8  into  the 
configuration  shown  in  Figure  3-9.  The  reference  voltages  (VRe^ (1) . . *vRef (I)) 
shown  in  Figure  3-9  represent  all  the  reference  voltage  levels  employed  by  a 
parallel  A/D  converter.  The  number  and  values  of  the  reference  voltages  VRe^(i) 
can  be  related  to  the  A/D  maximum  input  voltage  (Vu  ) and  its  least  significant 

rlclX 

quantum  level.  The  number  of  reference  voltages  required  is  equal  to  I ■ V / 

ntx 

VRef(l).  The  relationship  between  different  reference  voltages  can  be  expressed 


VR.f(1)  * VR.f(1-k>  + K VR.f(1) 

where  1-k  1 and  1 < I.  For  a given  input  signal  (Vg)  into  the  A/D  converter 
(where  |VRef(i)J  < {^j  < |VRef  ^ + *1  { ) the  output  (Q(i))  of  the  A/D  converter 
shown  in  Figure  3-9  will  be  as  follows. 


i 
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Q(i) 


[0  for  K + 1 f i j I 
1 for  i < K + 1 


Such  an  output  code  for  the  analog  Input  Vg  is  commonly  referred  to  as  a thermometer 


code.  This  thermometer  code  output  can  be  changed  into  a code  Q(i)  where  only 

the  Q(i)  equal  to  1 with  the  lowest  index  1 in  the  thermometer  code  will  remain 

*•  . 
a 1 (l.e.f  Q(i)  - Q(l)  and  all  other  Q(K)  will  be  equal  to  zero  for  K r i» 

Such  a transformation  from  the  thermometer  code  to  the  Q(l)  code  can  be 

implemented  by  a group  of  logic  blocks. 


♦ Q ( I ) 


♦ O (II) 


■♦0(1 


76  1030  VA  23 
Figure  3-10.  Thermometer  Code 

Each  logic  element  will  have  two  inputs  and  a single  output  as  shown  in  Figure  3-11. 


Q (.♦I)' 
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Figure  3-11.  Logic  Element 

The  logic  block  is  simply  two  input  AND  gate  where  only  one  input  is  Inverted. 

A group  of  these  logic  elements  can  readily  convert  the  Q(l)  to  the  Q(i) . The 
Q(l)  can  be  converted  into  a BCD  code  by  conventional  methods.  The  next  subject 
of  discussion  in  the  Median  Filter  is  formation  of  the  Histogram. 


i 


j 

I 
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3.3.2  Histogram 


A signal  processing  structure  will  be  discussed  for  extracting  different 
threshold  levels  from  a data  ensemble.  The  proposed  approach  is  easily 
adaptable  for  extraction  of  a median  from  a data  ensemble.  Moreover,  with 
some  modifications,  a time  varying  median  of  N data  elements  located  in  a 
moving  window  can  be  extracted.  Consider  five  data  elements  (see  Figure  3-12) 
represented  by  a ten  level  thermometer  code  stored  in  a bank  of  Shift  Register  (S/R) 
CCD  registers. 


10  Input 
Channels 


10  Output 
Channels 


76  1030  VA  25 

Figure  3-12  . Storage  Block  for  5 Data  Elements 
(el»  e2»  e3»  %»  and  e5* 

The  illustrated  bank  of  S/R  or  CCD  (l.e.,  memory  bank)  has  ten  parallel  inputs 
and  ten  parallel  outputs.  Clearly  the  median  of  the  data  elements  e^,  e 2,  e^, 
e^,  e5  is  the  e^  element.  Automatic  extraction  Of  the  median  can  be  achieved 
if  the  five  elements  shown  in  Figure  3-12  are  first  arranged  according  to  size 
and  the  middle  element  is  read  out  from  the  median  extracting  module.  Mecha- 
nization of  median  extraction  can  be  readily  achieved  if  the  clock  for  the  Rth 
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row  of  the  S/R  or  CCD  bank  (shown  in  Figure  3-12 ) is  slaved  via  a logic  function 
to  the  signal  appearing  at  the  R**1  input  channel.  In  Figure  3-13  the  clock  of 
the  memory  module  is  slaved  to  the  input  signal  via  an  AND  gate.  If  the  input 
signal  at  the  input  channel  is  a one,  the  clock  to  the  Memory  Module  is  enabled 
and  a one  will 


I • 
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Figure  3-13 . Clock  to  Memory  Module  (MM)  Slaved 
by  an  AND  Gate  to  Input  Signal. 

be  clocked  into  the  memory.  Otherwise  (i.e.,  when  the  input  is  a aero)  the 
clock  driving  the  memory  module  will  be  disabled  and  no  information  will  be 
clocked  into  the  memory  module.  Under  the  clock  constraints  outlined  for  the 
memory  module  (see  Figure  3-13)  a one  signal  will  appear  in  any  S^  location 
(i.e.,  Sj,  S^,  S^,  S^.  S^)  only  if  all  the  preceding  locations  with  a sub- 
script larger  than  i^  are  filled  with  a one  signal. 


Consider  ten  memory  modules  shown  in  Figure  3-12  assembled  into  a memory 
bank  (see  Figure  3-14)  equal  in  size  to  the  storage  block  shown  in  Figure  3-12. 
If  we  proceed  to  fill  the  memory  bank  shown  in  Figure  3-13  with  the  elements 
located  in  the  storage  block  shown  in  Figure  3-11  the  elements  e5,  e^,  e3,  e2 
and  e^  will <be  arranged  in  ascending  order  from  right  to  left. 
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Figure  3-14 . Memory  Bank  Consisting  of  10  Memory  Modules 
(Median  Extracting  Module) 

The  ordering  of  the  elements  e e^,  e^,  e^  and  e^  occurs  because  of  the  manner 
in  which  the  clock  of  each  memory  module  is  slaved  to  the  input  signal.  For 
example,  the  first  element  e^,  will  be  shifted  into  the  memory  bank's  lowest 
four  memory  modules  into  the  position.  Next,  the  e ^ element  will  appear 
at  the  memory  bank's  input.  Only  the  lowest  two  memory  module's  clocks  will 
be  clocked  hence  the  loaded  elements  e^  and  will  be  ordered  in  the  memory 
bank  such  that  e^  will  be  located  on  the  left  of  • Continuing  with  the 
elements  e^,  e^  and  e,.  it  is  easy  to  show  that  all  the  elements  will  be 
ordered  in  the  memory  block.  Clearly  if  nondestructive  taps  are  placed  on 
the  middle  (l.e.,  location  S^)  of  the  memory  bank  the  median  value  of  the 
elements  e^,  e^,  e^,  e^  and  e^  can  readily  be  obtained.  Moreover,  if  the 
elements  exceeding  the  20Z  threshold  level  are  sought  a nondestructive  read  out 
structure  attached  to  the  position  of  the  memory  bank  will  yield  this  level. 
The  size  of  the  memory  bank  (in  quantized  levels  and  data  elements)  can  be 
readily  extended  to  accommodate  larger  data  sets.  Similarly  levels  other  than 
the  median  can  be  readily  extracted. 


i 
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The  method  for  finding  a median  of  a data  ensemble  has  been  outlined.  The 
composition  of  the  outlined  median  extracting  module  (MEM)  is  such  that  a 
complete  data  ensemble  has  to  be  loaded  into  the  memory  bank  (see  Figure  3-14 ) 
before  the  median  is  obtained.  However  with  some  modifications,  the  median 
extracting  module  can  yield  the  median  of  N elements  selected  by  a moving 

'1 

window  N elements  wide.  In  this  operation  the  difference  between  successive 
sets  of  elements  located  within  the  moving  windows,  one  element  time  apart, 
will  be  a single  element.  It  is  desirable  to  compute  the  median  of  each 
successive  element  set  by  utilizing  the  signal  processing  already  performed 
on  the  N-l  elements  in  common  between  the  current  element  set  and  its  prede- 
cessor.  This  can  be  performed  if  we  simply  remove  from  the  median  extracting 
module  the  "old"  element  and  replace  it  with  the  "new"  element.  Mechanization 
of  this  requires  several  modifications  to  the  original  median  extracting  module* 
knowledge  of  the  "old"  and  "new"  elements  in  the  element  data  set , and  a 
memory  module  with  a slaved  clock  which  can  shift  the  stored  bits  left  or  right. 

The  "old"  element  can  be  obtained  if  an  additional  delay  storage  block  (e.g., 

see  Figure  3-12)  is  used  to  delay  the  signals  applied  to  the  input  of  the 

median  extracting  module.  The  size  and  delay  time  provided  by  the  delay 

storage  block  should  be  equal  to  the  size  and  delay  time  of  the  median 

extracting  module.  Hence  data  which  are  concurrently  fed  into  the  median 

extracting  module  and  the  delay  storage  block  (N  elements  wide)  will  emerge 

out  of  the  storage  block  in  phase  with  the  moving  window  N elements  wide.(See  Fig.  3 -12i 

Particularly  the  "old"  element  will  be  emerging  from  the  delay  storage  block 

when  the  "new"  element  will  be  entering  the  median  extracting  module  and  the 

delay  storage  block.  Knowing  the  "old"  and  "new"  elements ,the  clocks  for 

each  memory  module  (e.g.,  see  Figure  3-13),  composing  the  median  extracting 
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3-12a.  Block  Diagram  for  Moving  Window  Median  Extracting  Module 


module,  can  be  slaved  to  remove  the  old  element  and  insert  the  new  element  Into 
the  MEM.  The  function  by  which  the  clock  of  the  MM  will  be  governed  is  referenced 
in  Figure  3-12a  by  the  Clock  Cont’.  jI  Module  (CCM)  and  can  be  mathematically  expressed 


as  follows: 

(New)1  J+N 

- l 

Clock  disabled 

<01d>u 

VJ  (New)1>J+N 

* 0 

Clock  disabled 

(01d)iJ 

fl  (New)i,J+N 

- l 

Clock  shift  left 

n 

- l 

Clock  shift  right 

The  subscripts  (ij)  refer  respectively  to  the  i input  row  of  the  MEM  and  the 

second  subscript  refers  to  the  particular  element  from  the  ensemble.  Note 

that  the  "New"  and  "Old"  elements  are  separated  by  N data  elements.  Equation 

1,  governing  the  clock,  can  be  described  as  follows:  if  the  (Old)  . and 

l * J 

(New).  bits  are  equal  no  shifting  is  necessary,  when  the  (Old).  . bit  is 
1*J+N  1|J 

greater  than  the  (New).  . ...  bit  value  the  MM  should  shift  left  and  for  the 

i*J+N 

condition  when  the  (Old)  . . bit  is  less  than  the  (New).  . ...  bit  the  clock 

i»j  i»j+N 

should  shift  the  MM  right.  Embodiment  of  the  clock  control  slowed  to  the 

(New).  ....  and  (Old).  . input  can  be  realized  as  shown  in  Figure  3-15.  Two 
i»j+N  i.j 

different  clocks  are  necessary  for  shifting  the  MM  left  or  right  depending 
on  the  input  signals. 

At  the  end  of  each  line,  the  median  array  accumulates  the  last  columns 
based  on  the  old  row  and  the  first  columns  based  on  the  new  row.  Under  this 
mixed  condition,  median  calculations  are  suspended. 

Both  We8tlnghouse  and  Hughes  have  approached  the  histogram  problem,  based 
on  semiannual  reports,  from  an  analog-digital  hybrid  direction.  However,  the 
Westlnghouse  implementation  seems  more  able  to  accommodate  a moving  window  as 
found  in  the  Median  Filter. 
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4.0  FUTURE  DIRECTIONS 

In  line  with  our  emphasis  on  focal  plane  processing,  we  want  to  continue 
the  work  on  analog  Implementation  of  the  histogram. 

Also,  we  want  to  examine  the  cost  to  fabricate,  yields,  and  form  of  a 
possible  configuration  for  the  Gradient  Operator  of  the  Threshold  Algorithm. 

We  shall  be  examining  the  effectiveness  of  algorithms  such  as  Median  Filter 
within  the  context  of  an  entire  cueing  system. 

A final  item  is  to  analyze  the  proposed  line  growing  techniques  of 
Maryland  for  possible  implementation  by  CCD's. 
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